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This article reports on the effectiveness of an intervention using conceptual change approaches 
within challenging tasks, on the mathematics gains for low-performing year 3-6 students in in 
six primary schools. Quantitative data from PAT-Maths testing for each year showed a 
consistently large effect size of 0.7 compared to expected gain data from DECD. All six 
experimental groups caught up with DECD expectations within one year. Over the two years, 
students from years 3-5 gained an additional 27 months of mathematics learning over the 
expectations and students from years 4-6 gained 29 months, indicating the potential of the 
approach for closing educational gaps for low-performing students. 


In recent decades, the assessment revolution (Broadfoot and Black, 2004) has contributed 
to a prioritising of quantitative data for accountability and predictability, and subsequently, an 
emphasis on testing. As the pressure on teachers to produce high results has increased (Gaffney 
& Faragher, 2014), concerns have been raised in Australian schooling regarding effective 
intervention strategies for closing the educational gap for low-performing and at-risk students 
(Ewing, 2011; Masters, 2009). 

One strategy often recommended to close educational gaps for low-performing students is 
direct instruction (Ewing, 2011; Farkota, 2003; Masters, 2009). Direct instruction is relatively 
simple to implement (Ewing, 2011), however critics argue that an instructivist approach leads 
to overemphasis on memorisation of procedures rather than conceptual understanding (Cooney, 
2001). Conversely, grappling with challenging tasks supports conceptual thinking and making 
connections in mathematics, thus producing greater overall learning gains (Boaler & Staples, 
2008; Stein & Lane, 1996). However, research literature regarding the use of challenging tasks 
with low-performing students indicates a strong tendency for teachers to reduce the cognitive 
demand in tasks when they perceive that students may struggle (Archambault et al., 2012; Stein 
& Lane, 1996). 

Conceptual change programs, which are gaining momentum for their high impact on 
student learning in science classrooms (Hattie, 2015), may provide a way for teachers to address 
student difficulties within challenging mathematics tasks without reducing the cognitive load 
(Kennedy, 2015a). While research literature regarding the application of conceptual change 
approaches in mathematics classrooms is limited (Swan, 2001), research by Kennedy (2015a) 
connected lesson structures developed for conceptual change in science (Erilymaz, 2002) with 
highly-regarded lesson structures for challenging tasks in mathematics (Lapan et al., 2006) to 
develop a model for using conceptual change questioning when students struggle in challenging 
tasks. 

The current study used quantitative data gathered on standardised tests over a two-year 
period to explore the effectiveness of an intervention based on Kennedy’s (2015a) model. The 
research explored the improvement in learning and the effect of the intervention at “closing the 
educational gap” (Ewing, 2011, p.66) for low-performing year 3-6 students in six South 
Australian primary schools. 


2018. In Hunter, J., Perger, P., & Darragh, L. (Eds.). Making waves, opening spaces (Proceedings of the 
41* annual conference of the Mathematics Education Research Group of Australasia) pp. 447-454. 
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Literature review 


The move towards greater accountability in recent decades (Gaffney & Faragher, 2014) has 
led to trialling of intervention strategies to improve learning gains (Farkota, 2003) and “close 
the educational gaps” (Ewing, 2011, p.66) for low-performing students. In a review of 
numeracy and literacy achievement across Australian schools, Masters (2009) found that by 
year five a gap of 2.5 years was observable between the lowest 20% and highest 20% of students 
and increased with each subsequent year of schooling. This observation led Masters to conclude 
that, “Australian students who slip behind in their literacy and numeracy learning during their 
primary years often never catch up.” (Masters, 2009, p.vii). 

Reports on Australian schooling recommend direct instruction as an effective intervention 
for low-performing students (Ewing, 2011; Farkota, 2003; Masters, 2009). With an effect size 
(Cohen’s d, Cohen, 1988) of 0.60 in standardised testing (Hattie, 2015), direct instruction is an 
appealing strategy for school leaders who feel pressured to improve student results in such 
testing (Masters, 2009), thus setting the benchmark by which other interventions may be judged 
(Ewing, 2011). 

In contrast, while research on challenging tasks has found large overall learning gains for 
students (Boaler & Staples, 2008; Stein & Lane, 1996), the type of evidence gathered does not 
easily align with the quantitative data produced by standardised testing. Australian research into 
challenging tasks has tended to focus on student engagement in questions that are authentic and 
complex rather than simple or standardised, such as those that require making connections 
between concepts, devising solution strategies and exploring multiple pathways to solutions 
(Sullivan et al., 2006; Sullivan et al., 2013; Sullivan et al., 2014). This presents a significant 
dilemma for schools and teachers who wish to trial challenging tasks yet are increasingly judged 
on improved student achievement from standardised testing (Gaffney & Faragher, 2014; 
Masters, 2009). 

Concerns have also been raised regarding how teachers use challenging tasks with low- 
performing students. Researchers have observed that teachers have difficulty identifying 
potential barriers to learning prior to engaging in challenging tasks (Mousley et al., 2007), may 
be unsure how to address student misconceptions and alternative conceptions that are uncovered 
(Son & Kim, 2015), and tend to reduce the cognitive demand in tasks when they perceive 
students may struggle (Archambault et al., 2012; Stein & Lane, 1996). Creating a class culture 
of persistence (Sullivan et al., 2013), teacher questioning in response to correct or incorrect 
answers (Boaler & Staples, 2008; Swan, 2001), and the use of enabling prompts to maintain the 
cognitive load (Sullivan et al., 2006), have been identified as key components for supporting 
students who struggle in challenging tasks. 

Conceptual change programs may provide a way of addressing the concerns raised by 
researchers with using challenging tasks for low-performing students (Kennedy, 2015a), while 
still producing the quantitative data valued within the current climate of the assessment 
revolution. Research on the impact of conceptual change programs in science classrooms have 
consistently produced large effect sizes of 1.16 (Hattie, 2015), indicating significant potential 
for closing educational gaps. When applying a similar approach to mathematics, Kennedy 
(2015a) found that incorporating conceptual change questioning into challenging mathematics 
tasks encouraged students to alter their existing conceptions. 

Conceptual change approaches centre around the idea of motivating learners to change their 
own ideas or beliefs (Mayer, 2008; Posner et al., 1982; Resnick, 1983). They typically involve 
using a challenging situation or problem paired with discrepant events and questioning 
(Erilymaz, 2002; Swan, 2001) to create cognitive conflict (Posner et al., 1982; Resnick, 1983) 
whereby a learner recognizes an anomaly in his or her own thinking and then actively constructs 
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a new model that explains the observable facts (Mayer, 2008). In an analysis of 20 incidents of 
conceptual change within challenging mathematics lessons, Kennedy (2015a) identified the 
following common phases: 


Launch: Engaging learners with a challenging mathematical problem and identifying 
alternative conceptions through their initial responses. 

Exploration: Experimenting with learners’ own ideas to solve the problem. When 
alternative conceptions were identified, the teacher juxtaposed discrepant events with 
questioning, until cognitive conflict was observed. 

Accommodation: Cognitive conflict was increased with questioning and discrepant events 
until learners recognised that their existing ideas were anomalous or inadequate. At this 
point learners were observed to change their own minds and develop a new idea or 
conception that better explained the observable facts. 

Resolution: Learners solved the initial challenging problem using their new idea. 

Generalisation: Learners applied the new conception to solve a challenging problem. 


The current study explores the effectiveness of an intervention based on Kennedy’s (2015a) 
model for conceptual change in challenging tasks, on the learning gains for low-performing 
students. In acknowledging the current emphasis in Australian schools on improving the gains 
for low-performing students on standardised testing (Masters, 2009), data presented in this 
paper are drawn from year 3-6 classrooms from six primary schools over a two-year period 
using the Progressive Achievement Tests in Mathematics (PAT-M) developed by the Australian 
Council for Educational Research (ACER). 

This paper is underpinned by several assumptions. The first assumption is that almost all 
students are capable of succeeding at mathematics to high levels (Askew et al., 1997). Second 
is the assumption that effective teaching can be defined by learning gains for students (Askew 
et al., 1997), and that in the end, schools are judged by increases in student achievement 
(Gaffney & Faragher, 2014). A third assumption is that for learning to be effective, students 
need to think deeply, connect ideas and be challenged (Boaler & Staples, 2008). The final 
assumption is that when new information conflicts with learners’ existing conceptual 
understanding, it is either rejected outright or accommodated by changing the underlying 
conceptions (Posner et al., 1982; Resnick, 1983). 


Method 


Context 


Leaders from six public primary schools in regional South Australia requested a 
professional learning project to improve learning gains for their students. Three of the schools 
were medium sized (pop. 100-300), three were small (pop. 60-100) and all were located within 
50km of Adelaide. Leaders cited concerns about their students’ performance on PAT-M and 
NAPLAN testing. They noted that student data from their first year of PAT-M testing (2015) 
were lower than expected, and that their numeracy data from 2015 NAPLAN testing showed 
low cohort gain for years 3-5 (see Results for more detail). 


Participants 


All year 3-6 teachers and students from the primary schools participated in the study. For 
the purpose of this report, data are limited to the learning gains made by the lowest 20% of 
students by pre-test scale score, in line with the approach taken in Masters’ (2009) report. Each 
of the primary schools participated in annual online PAT-M testing from September-October 
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in 2015, 2016 and 2017 according to protocols set by South Australia’s Department for 
Education and Child Development (DECD). All tests were marked externally by ACER and 
the data were returned to each school for further analysis. At the school level, data were 
examined to identify all students who participated in any two consecutive tests. These data 
points were deidentified, then grouped by year level for 2015-2016 and 2016-2017, forming six 
groups for further analysis (see Table 1). 


Professional learning model 


Helping teachers to reconsider and change their own beliefs is considered an important 
element in effective professional learning (Beswick, 2008). Within this project a conceptual 
change lens was applied to professional learning, for its potential to encourage teachers to 
reconsider and alter their own ideas or beliefs (Mayer, 2008) thereby influencing their practices 
(Hawley & Valli, 2000). Throughout eight days of live professional learning, leaders and key 
teachers from each school both raised and engaged with problems they had experienced with 
classroom teaching exposing their beliefs regarding the nature of mathematics, the nature of 
learning, the nature of problem-solving and the nature of teaching (see Askew et. al, 1997). 
Discrepant events such as observing modelled lessons using conceptual change, predicting and 
then reflecting on student responses, and trying out the teaching approach with students were 
used to help participants to reconsider and change their own beliefs, before they trained their 
remaining teachers. All teachers were provided with support from their leadership, two days of 
live professional learning with the researcher, an 18-hour sequence of webinars, and a set of 
adaptable lesson plans created by the researcher (Kennedy, 2015b) as a starting point for 
implementing conceptual change approaches. In the first year of the study the prepared lessons 
were used for 1-2 lessons per week by teachers, however, in the second year the teachers moved 
towards creating their own lessons using a similar structure. 


Data analysis 


PAT-M testing was selected for this study as it provided objective and norm-referenced 
information on students’ level of achievement, their skills, and understanding of mathematics 
(Lindsey, Stephanou, Urbach, & Sadler, 2005), and as data gathered from PAT-M testing 
formed the basis for Australian research on the impact of direct instruction (Farkota, 2003). In 
the current study, published standards of expected achievement (SEA) developed by DECD 
(DECD, 2016) were substituted for control groups due to ethical and practical considerations, 
with effect size calculated using Cohen’s d (Cohen, 1988). 

To help quantify the impact of the intervention on closing the gaps for low-performing 
students, effect sizes for all experimental groups were also calculated in terms of months of 
additional mathematics gain over that which was expected in the SEA. As advance in 
achievement tends to change with students’ year levels (Lee et al., 2012) the time-indexed 
approach to effect size for mathematics interventions developed by Lee et al. (2012) was 
applied to calculate months of additional gain. As an additional control, cohort gain data 
obtained in NAPLAN for the two years immediately prior to the study (2013-2015) were 
compared with data obtained throughout the study (2015-2017). 


Results 


Data are presented below to examine the effect of the intervention on the PAT-Maths 
growth in mean scale score for each experimental group in comparison to the DECD SEA. For 
each experimental group, data from paired t-Tests are presented to demonstrate statistical 
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significance, along with the means, standard deviations, and effect sizes. Table J presents pre- 
and post-test comparison data for each experimental group, control data from the SEA and the 
effect size of the experiment. Standard deviation for each experimental group was calculated 
using pooled data across the pre- and post-test results. All experimental groups made 


statistically significant gains, with P figures at or below 0.001. 


Table 1 
Characteristics of Experimental Groups and PAT-M Data, by Year Level 
Year 3-4 Year 4-5 Year 5-6 
2015- 2016- 2015- 2016- 2015- 2016- 
2016 2017 2016 2017 2016 2017 
n 22 21 17 2A 20 16 
Experimental group gain 14.7 12.9 6.9 7.6 9.0 10.7 
(pre-test and post-test means) | (96.8- (98.1- | (108.1- | (104.4- | (113.9- | (111.1- 
111.6) | 111.0) | 115.0) | 111.9) | 122.9) | 121.8) 
DECD SEA expected gain@ 9 (101-110) 2 (110-112) 8 (112-120) 
Standard deviation 6.37 5.29 4.85 7.03 4.99 5.58 
t Stat -8.21 -8.46 -3.47 -4.15 -6.15 -7.70 
Effect size (Cohen’s d) 0.90 0.74 1.01 0.79 0.21 0.49 
Gain in months 14 10 16 13 5 13 


4 DECD Standard of Expected Achievement Gain Data (DECD, 2016) 


The first year of the project produced a large mean effect size of 0.70 across all experimental 
groups. This was consistent in 2016-2017 (d = 0.67). Data pooled across the two years of the 
project showed an additional gain of 27 months for years 3-5 students over the SEA, and an 
additional gain of 29 months for years 4-6 students. Figure / illustrates the gains in shown in 
Table I for each experimental group, compared with expected gains. In five out of the six 
experimental groups the pre-test means were lower than SEA recommendations. In all cases, 
the post-test means for the experimental groups were the equal to or higher than the SEA and 
growth exceeded the SEA. 


Actual vs. Expected Learning Gains on PAT M Testing, 
for 2015-2016 and 2016-2017 by Year Level 
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Figure 1. Learning gains by experimental group vs. expected learning gains from 
DECD SEA in scale score for PAT-M testing, 2015-2016 and 2016-2017. 
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To check the validity of the high effect size in PAT-M data, additional control data from 
NAPLAN testing for the two-year period prior to the study were compared with data obtained 
during the study. Table 2 shows the comparative NAPLAN cohort gain data for the research 
cohort with the cohort gain for all South Australian schools. Due to the small size of three of 
the schools, some cohort gain data were not available for comparison. Each available school’s 
cohort gain was calculated by subtracting the mean scale score of the pre-test from the post-test 
for the stated time-period, then compared with the same calculation made using publicly 
available data on all South Australian schools from the NAP website. Table 2 below shows that 
the percentage of schools in the study meeting or exceeding SA cohort gain for years 3-5 rose 
from 60% to 83% over the period of the study. This improvement in control data from 
NAPLAN supports the validity of the PAT-M data. 


Table 2 

Comparing year 3-5 Cohort gain data in NAPLAN prior to and during the study 
Percentage of schools that met the 2013-2015 SA gain@ 3 out of 5 schools (60%) 
Percentage of schools that met the 2015-2017 gain@ 5 out of 6 schools (83%) 


4Cohort gain calculations were made using data from the MySchool and NAP websites. 


Discussion 


The research reported in this article explored the effect of an intervention using Kennedy’s 
(2015a) model for integrating conceptual change within challenging problems on the 
mathematics learning gains made by low-performing students in six SA primary schools. The 
primary finding was that the student achievement improved significantly above the expected 
levels on standardised testing, thus closing the educational gap faced by low-performing 
students. 

The intervention reported on in this study produced a high effect size of 0.7 over and above 
the annual gain expected for PAT-M provided in the DECD SEA, for the lowest 20% of 
students. This was consistent across both years of data collection. As teachers are under pressure 
to improve student results on standardised testing (Gaffney & Faragher, 2014) PAT-M data 
analysis was selected for this study to enable comparison with existing research. The effect size 
of the intervention reported on in this study exceeded the benchmark of 0.60 set by direct 
instruction (Ewing, 2011; Hattie, 2015). 

This study sought to quantify the effectiveness of the intervention on closing the educational 
gap experienced by low-performing students. Masters (2009) reported that the educational gap 
between the lowest and highest 20% of year five students was 2.5 years. Data from this study 
indicate that the lowest 20% of students from years 3-5 gained an additional 27 months over 
the expected growth, and the lowest 20% of students from years 4-6 gained an additional 29 
months over the expected growth (see Lee et al., 2012). Over the course of the study all 
experimental groups were observed to catch up with the recommended SEA for PAT-M within 
12 months, with the growth for all groups significantly exceeding the recommendations. These 
findings indicate significant potential for conceptual change interventions based on Kennedy’s 
(2015a) model, in closing educational gaps for low-performing students. 


Limitations 


This study has several limitations. In terms of participants, this study contains data from a 
relatively small number of students, all from small to medium primary schools in regional South 
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Australian schools. A larger study, with more varied schools, is recommended to confirm the 
findings. A second limitation is that baseline data for PAT-M gain were not available for this 
study, so control data from both the DECD SEA and from NAPLAN were used to compensate 
and to check the reliability of the growth. It would be useful to repeat the study with a larger 
group of students and teachers for whom baseline PAT-M gain data are available to confirm 
the findings. A final limitation is that the researcher used her own resources for part of this 
study as other research and resources on conceptual change approaches in mathematics are 
limited. To manage this potential conflict of interest, all data were collected, analysed and 
reported using external sources from ACER (PAT-M), the Australian Curriculum and 
Assessment Authority (NAPLAN) and DECD (SEA). All data are available on request. Future 
reports derived from the current study are expected to report on the impact of the professional 
learning model on teacher beliefs and practices, as well as examining the effectiveness of 
interventions based on Kennedy’s (2015a) model on the learning gains made by high- 
performing students. 


Conclusion 


The results of this study indicate that standardised mathematics testing results for low- 
performing students from years 3-6 improved when an intervention based on Kennedy’s 
(2015a) model for conceptual change within challenging tasks was implemented for 1-2 lessons 
per week over a two-year period. A large and statistically significant effect of 0.7 was observed 
across each experimental group, for each year of the project. Over the course of the study, all 
experimental groups caught up with recommendations provided by DECD for PAT-M, with 
low-performing students from years 3-5 gaining an additional 27 months over the expected 
growth, and low-performing students from years 4-6 gaining an additional 29 months over the 
expected growth. This suggests that Kennedy’s (2015a) model may provide a viable 
intervention for improving the learning gains and addressing the education gaps experienced 
by low-performing students. 
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